-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CASSANDRA-19664 first approach - for copying Basic Table definition #3718
base: trunk
Are you sure you want to change the base?
Conversation
@@ -236,6 +236,7 @@ cqlStatement returns [CQLStatement.Raw stmt] | |||
| st42=addIdentityStatement { $stmt = st42; } | |||
| st43=dropIdentityStatement { $stmt = st43; } | |||
| st44=listSuperUsersStatement { $stmt = st44; } | |||
| st45=copyTableStatement { $stmt = st45; } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason i used a new statement here is that :
1、The original createtablestatment code is too bloated. If the semantics of like are added, the code needs to be changed a lot.
2、In addition, I think this new copystatement can do things other than table cloning, and can also support semantic capabilities such as “CREATE TABLE AS”. Because they are similar to table copying functions, they are called copystatement here.
Besides, I have also prepared a patch, which is mainly to modify the pares of create table, that is, to implement the function of create table like in create table statement. If you think it is necessary, I can create a new PR to replace the current one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is another pr here that implement CASSANDRA-19964, but use create table's parase path .
I am not sure which one is better , so I attach both of them.
52f4d30
to
8b09b40
Compare
8b09b40
to
b826a62
Compare
throw ire("Cannot use CTREATE TABLE LIKE across different keyspace when source table have UDTs."); | ||
} | ||
|
||
String sourceCQLString = sourceTableMeta.toCqlString(false, false, true, false); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we ignore the dropped columns, if the seconds input argument is true that means the dropped column will be consider .
Then the cql will be like : create table tb (pk int primary key, cn int, dn int) with xxxx ; alter table tb drop dn;
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Outdated
Show resolved
Hide resolved
} | ||
|
||
@Override | ||
public void validate(ClientState state) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we need this? Will not be validate
of super called automatically when you do not override?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes , we can remove it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You also need to do this, no?
https://github.com/apache/cassandra/pull/3721/files#r1871469974
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Maxwell-Guo please cover my last comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi , I replied here, I think we may not need to do this , because the CQLs that are parsed are all create table like syntax, there will be no coexistence with create table syntax. For cql like create table if not exist ta like tb
or some other cql that contains these three keywords : CREATE / TABLE / LIKE , only create table a like b
can be successfully paresd into copytablestatemnt, others will got a SyntaxException. I have a test for this in CreateLikeCqlParseTest.
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Outdated
Show resolved
Hide resolved
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Outdated
Show resolved
Hide resolved
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Outdated
Show resolved
Hide resolved
173c775
to
e40726c
Compare
test/unit/org/apache/cassandra/cql3/validation/entities/CollectionsTest.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is my first round of review. :-) Thanks a lot for pushing that functionality forward.
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Show resolved
Hide resolved
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Outdated
Show resolved
Hide resolved
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Outdated
Show resolved
Hide resolved
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Outdated
Show resolved
Hide resolved
String souceTable = createTable(KEYSPACE_PER_TEST, | ||
"CREATE TABLE %s (" + | ||
" pk1 text, " + | ||
" pk2 int MASKED WITH DEFAULT, " + | ||
" ck1 int, " + | ||
" ck2 double," + | ||
" s1 float static, " + | ||
" v1 int, " + | ||
" v2 int, " + | ||
"PRIMARY KEY ((pk1, pk2), ck1, ck2 ))"); | ||
String targetTable = createTableLike("create table %s like %s", souceTable, KEYSPACE_PER_TEST, KEYSPACE_PER_TEST); | ||
Pair<TableMetadata, TableMetadata> pair = assertTableMetaEquals(KEYSPACE_PER_TEST, KEYSPACE_PER_TEST, souceTable, targetTable); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It could be replaced by:
String souceTable = createTable(KEYSPACE_PER_TEST,
"CREATE TABLE %s (" +
" pk1 text, " +
" pk2 int MASKED WITH DEFAULT, " +
" ck1 int, " +
" ck2 double," +
" s1 float static, " +
" v1 int, " +
" v2 int, " +
"PRIMARY KEY ((pk1, pk2), ck1, ck2 ))");
TableMetadata sourceTableMetadata = currentTableMetadata();
String targetTable = createTableLike("create table %s like %s", souceTable, KEYSPACE_PER_TEST, KEYSPACE_PER_TEST);
TableMetadata targetTableMetadata = currentTableMetadata();
assertTableMetaEquals(sourceTableMetadata, targetTableMetadata);
and assertTableMetaEquals
can be moved to DescribeStatementTest
and replaced by:
protected void assertTableMetaEquals(TableMetadata source, TableMetadata target) {
assertNotNull(source);
assertNotNull(target);
assertTrue(source.equalsWithoutTableNameAndDropCns(target));
assertNotEquals(source.id, target.id);
assertNotEquals(source.name, target.name);
}
It will then allow to remove the getTableMetadata
methods in CQLTester.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
assertTableMetaEquals
will aslo be used for other test classes like
CreateLikeTest -> testTableShemaCopy ()
- I remove the
getTableMetadata
in CQLTester - I change the test code in DescribeStatementTest and remove the use of assertTableMetaEquals, and verify them one by one :
assertNotNull(source);
String targetTable = createTableLike("create table %s like %s", souceTable, KEYSPACE_PER_TEST, KEYSPACE_PER_TEST);
TableMetadata target = getTableMetadata(KEYSPACE_PER_TEST, currentTable());
assertNotNull(target);
assertTrue(equalsWithoutTableNameAndDropCns(source, target));
assertNotEquals(source.id, target.id);
assertNotEquals(source.name, target.name);
That is because
(1) currentTableMetadata();
only return the table meta under KEYSPACE;
(2) the assertTableMetaEquals
is only used once, but it is used many time in CreateLikeTest
so I move assertTableMetaEquals
to CreateLikeTest
,
WDYT ? @blerer
protected TableMetadata getTableMetadata(String table) | ||
{ | ||
return Schema.instance.getTableMetadata(KEYSPACE, table); | ||
} | ||
|
||
protected TableMetadata getTableMetadata(String keyspace, String table) | ||
{ | ||
return Schema.instance.getTableMetadata(keyspace, table); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can be removed after the changes suggested in DescribeStatementTest
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see the comment on assertTableMetaEquals
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@blerer
I removed
{
return Schema.instance.getTableMetadata(KEYSPACE, table);
}
But kept
protected TableMetadata getTableMetadata(String keyspace, String table)
{
return Schema.instance.getTableMetadata(keyspace, table);
}
as getTableMetadata(String keyspace, String table)
is used by some test cases, is it ok ?
@@ -479,12 +479,21 @@ public String defaultCompactValueName() | |||
} | |||
} | |||
|
|||
public static TableMetadata.Builder parse(String cql, String keyspace, String table, Types types, UserFunctions userFunctions) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel that I am missing something. Why did you change that code?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@blerer the original version of that method (which calls this one) is still there. We just create this one in order to be able to specify custom table name with types and functions. If we parse CQL for old table, we just want to be able to put there table name etc for new table directly upon parsing.
0a7b1f5
to
ed700a8
Compare
ed700a8
to
bbe3ad6
Compare
protected TableMetadata getTableMetadata(String table) | ||
{ | ||
return Schema.instance.getTableMetadata(KEYSPACE, table); | ||
} | ||
|
||
protected TableMetadata getTableMetadata(String keyspace, String table) | ||
{ | ||
return Schema.instance.getTableMetadata(keyspace, table); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@blerer
I removed
{
return Schema.instance.getTableMetadata(KEYSPACE, table);
}
But kept
protected TableMetadata getTableMetadata(String keyspace, String table)
{
return Schema.instance.getTableMetadata(keyspace, table);
}
as getTableMetadata(String keyspace, String table)
is used by some test cases, is it ok ?
@Test | ||
public void testIfNotExists() throws Throwable | ||
{ | ||
String sourceTb = createTable(sourceKs, "CREATE TABLE %s (a int, b text, c duration, d float, PRIMARY KEY(a, b));"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@blerer I add one test case for the IF NOT EXISTS
as you suggested
assertTableMetaEquals(sourceKs, targetKs, tableOtherOptions, tbLikeCompactionOthers); | ||
|
||
// table copy with params setting | ||
String tableCopyAndSetCompression = createTableLike("CREATE TABLE %s LIKE %s WITH compression = { 'class' : 'SnappyCompressor', 'chunk_length_in_kb' : 64 };", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@smiklosovic I update the pr and implemented setting table params when doing CREARE TABLE LIKE
import org.apache.cassandra.transport.Event.SchemaChange; | ||
|
||
/** | ||
* {@code CREATE TABLE [IF NOT EXISTS] <newtable> LIKE <oldtable> WITH <property> = <value>} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@blerer this is a brief introduction of cql to CopyTableStatement, and I have already create CASSANDRA-20120 to create CEP-43's doc
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Show resolved
Hide resolved
src/java/org/apache/cassandra/cql3/statements/schema/CopyTableStatement.java
Show resolved
Hide resolved
test/unit/org/apache/cassandra/schema/createlike/CreateLikeTest.java
Outdated
Show resolved
Hide resolved
test/unit/org/apache/cassandra/schema/createlike/CreateLikeTest.java
Outdated
Show resolved
Hide resolved
test/unit/org/apache/cassandra/schema/createlike/CreateLikeTest.java
Outdated
Show resolved
Hide resolved
|
||
public class CreateLikeCqlParseTest extends CQLTester | ||
{ | ||
private static final String[] unSupportCqls = new String[] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Maxwell-Guo try to think about more cases which are not supported. Change "unSupportCqls" to "unsupportedCqls".
However, in general, the added value of this test is questionable as we have the grammar for this so I am not completely sure what we are actually testing here ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will add more unsupported test cql cases .
The main situation we want to test here is: the scenario where create table and create table like syntax are mixed.
@@ -296,6 +304,22 @@ public void testCounterStatement() | |||
assertGreaterThan(cfs.metric.coordinatorWriteLatency.getMeanRate(), 0); | |||
} | |||
|
|||
@Test | |||
public void testCreateLikeTableMetric() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Maxwell-Guo I am not completely sure what is the added value of this test. We can not just test absolutely everything again just because a table was copied. Why is this test necessary? If we copy a table, it is as any other, then it means that it will also behave like any other and we do need to test it for the second time.
You just need to test that after you copy a table, that table is equal to the source one, structurally, and is independent from the original one (e.g. modification / alterations of fields and changing options does not change the options on the original table).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your suggestion. The main purpose of adding this test here is to make the cloned table a black box. The table can be written and read normally, and the corresponding metric can be updated normally.
However, I have the test case you mentioned in CreateLikeTest, so I think this test can be deleted.
|
||
TableParams originalParams = targetBuilder.build().params; | ||
TableParams newTableParams = attrs.asAlteredTableParams(originalParams); | ||
TableMetadata table = targetBuilder.params(newTableParams) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Maxwell-Guo could you please test what happens when I do this?
CREATE TABLE ks.tb_copy LIKE ks.tb WITH ID = '....';
Can I provide a custom ID for my copied table?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will give you ID too:
DESCRIBE TABLE ks.tb3 WITH INTERNALS ;
You can then take that "WITH ID", increase it by one and create that "like" table and see what happens.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is supported in grammar level , cql can be parsed , but a ConfigurationException of "Cannot alter table id." will be thrown at apply level.
when doing :
TableParams newTableParams = attrs.asAlteredTableParams(originalParams);
and see :
TableParams asAlteredTableParams(TableParams previous)
{
if (getId() != null)
throw new ConfigurationException("Cannot alter table id.");
return build(previous.unbuild());
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need to find a way how to support this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not very clear about this, why do we need to support setting table id?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And if we want to support setting table id through "create table like " cql ,we can just add flag here to ignore this
TableParams asAlteredTableParams(TableParams previous, boolean ignore)
{
if (ignore && getId() != null)
throw new ConfigurationException("Cannot alter table id.");
return build(previous.unbuild());
}
`
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As create table support setting table id , I think we should also have this ability. I will make an update for this pr. Thanks @smiklosovic
This is for CASSANDRA-19664:
1、we can copy table under the same keyspace of under different keysapces.
2、only support basic table schema : primary key(partition key and clustering key), regular columns , static columns, column masking, table params(compaction, compression and so on).